There exists a general need in the development of software/systems to ensure that the finished product is sufficiently tested prior to its release to the public. One form of testing comprises submitting various combinations of values in accordance with a specified program interface (e.g., an application programming interface). In many, if not most, cases a large number of input parameters, and the many different potential values assignable to each of those input parameters precludes exhaustive testing of the program code. As a result, previous methods for testing software settled for less than complete coverage for program modules that receive a set of input parameter values.
One method of testing involves submitting all, or a random subset of all, conceivable bit combinations to a program interface and observing the operation and/or results of each unique input bit combination. Many, if not most, program interfaces accept several parameters potentially comprising a large number of total bits rather than a few parameters comprising a relatively small number of total bits. In cases where the results are manually observed/analyzed, the cost of adequately testing a program module becomes prohibitive. As an example of the impracticality of testing every conceivable bit combination, consider an API (application program interface) that receives four double word (2×16 bit) parameters for a total of 128 bits. Testing every input bit combination requires 2128, or two to the one hundred twenty-eighth power (an eternity) input bit combinations. Many API's accept even greater numbers of parameters/bits.
A number of alternative approaches to reducing (diluting) a test matrix, comprising a set of test input parameter value combinations to a program module, are known. Such approaches utilize a reduced number of parameter values, and in some cases as little as one value, from a class of values having the same effect upon the code (parameter value equivalence classes). However, the alternative combination reduction approaches, described further herein below, do not adequately address complex program interfaces that accept a large number of variables having complex co-relations and inter-dependencies. By way of example, the well known MICROSOFT API “MQSendMessage” has 192 parameters. Classes of particular input parameter values (“equivalence classes”) having similar effect upon the program module are defined, and a single value from each equivalence class is used to define/generate a test matrix providing minimal coverage for the potential input parameter value combinations. However, the total number of distinct input combinations still remains prohibitively large. Alternatively, some combinations are skipped to reduce the total number of test input parameter combinations, but such dilution introduces the possibility of missing an important combination.
A well-known test matrix generation method, “PICT,” adopts a dilution scheme that reduces coverage to value combinations of two parameters. The value combinations are diluted according to defined equivalence classes of parameter values. The success of this test matrix dilution scheme assumes that a program error will result in a failure over a certain combination of two parameter values. A simple example shows why this assumption is unfounded. Consider a function called calc(x,op,y) which returns the value of x<op>y. Setting x=0 is fine, setting y=0 is OK and setting op=^ is also good. But pair-wise testing does not guarantee coverage of the three-parameter combination 0^0. In fact, there's a good chance it never will: having covered 0^a, a^0(a≠0) and 0+0, there's no need (according to this solution) to check 0^0. Thus, the above pair-wise test generation/execution method does not ensure a program path associated with the input parameter values x=0, y=0 and op=^ is good.